v2.10.22

Compare Source

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.22.8

Dependencies

• golang.org/x/crypto v0.28.0 (#​5971)
• golang.org/x/sys v0.26.0 (#​5971)
• golang.org/x/time v0.7.0 (#​5971)
• go.uber.org/automaxprocs v1.6.0 (#​5944)
• github.com/klauspost/compress v1.17.11 (#​6002)

Added

Config

• A warning will now be logged at startup if the JetStream store directory appears to be in a temporary folder (#​5935)

Improved

General

• More efficient searching of sublists for the number of subscriptions (#​5918)

JetStream

• Improve performance when checking interest and correcting ack state on interest-based or work queue streams (#​5963)
• Safer default file permissions for JetStream filestore and logs (#​6013)

Fixed

JetStream

• Large number of message delete tombstones will no longer result in unusually large message blocks on disk (#​5973)
• The server will no longer panic when restoring corrupted subject state containing null characters (#​5978)
• A data race when processing append entries has been fixed (#​5970)
• Fix a stream desync across replicas that could occur after stalled or failed catch-ups (#​5939)
• Consumers will no longer race with the filestore when fetching messages, fixing some cases where consumers can get stuck with workqueue streams and max messages per subject limits (#​6003)
• Pull consumers will now recalculate max delivered when expiring messages, such that the redelivered status does not report incorrectly and cause a stall with a max deliver limit (#​5995)
• Clustered streams should no longer desync if a catch-up fails due to a loss of leader (#​5986)
• Fixed a panic that could occur when calculating asset placement in a JetStream cluster (#​5996)
• Fixed a panic when shutting down a clustered stream (#​6007)
• Revert earlier PR #​5785 to restore consumer start sequence clipping, fixing an issue where sourcing/mirroring consumers could skip messages (#​6014)

Leafnodes

• Load balancing of queue groups over leafnode connections (#​5982)

Complete Changes

v2.10.21

Compare Source

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.22.7

Dependencies

• golang.org/x/crypto v0.27.0 (#​5869)
• golang.org/x/sys v0.25.0 (#​5869)

Added

Config

• New TLS min_version option for configuring the minimum supported TLS version (#​5904)

Improved

JetStream

• Global JetStream API queue hard limit for protecting the system (#​5900, #​5923)
• Orphaned ephemeral consumer clean-up is now logged at debug level only (#​5917)

Monitoring

• statsz messages are now sent every 10 seconds instead of every 30 seconds (#​5925)
• Include JetStream pending API request count in statsz messages and jsz responses for monitoring (#​5923, #​5926)

Fixed

JetStream

• Fix an issue comparing the stream configuration with the updated stream assignment on stream create (#​5854)
• Improvements to recovering from old or corrupted index.db (#​5893, #​5901, #​5907)
• Ensure that consumer replicas and placement are adjusted properly when scaling down a replicated stream (#​5927)
• Fix a panic that could occur when trying to shut down while the JetStream meta group was in the process of being set up (#​5934)

Monitoring

• Always update account issuer in accountsz (#​5886)

OCSP

• Fix peer validation on the HTTPS monitoring port when OCSP is enabled (#​5906)

Config

• Support multiple trusted operators using a config file (#​5896)

Complete Changes

v2.10.20

Compare Source

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.22.6

Fixed

JetStream

• Fix regression in KV CAS operations on R=1 replicas introduced in v2.10.19 (#​5841) Thanks to @​cbrewster for the report!

Complete Changes

v2.10.19

Compare Source

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.22.6

Dependencies

• golang.org/x/crypto v0.26.0 (#​5782)
• golang.org/x/sys v0.24.0 (#​5782)
• golang.org/x/time v0.6.0 (#​5751)

Improved

General

• Reduced allocations in various code paths that check for subscription interest (#​5736, #​5744)
• Subscription matching for gateways and reply tracking has been optimized (#​5735)
• Client outbound queues now limit the number of flushed vectors to ensure that very large outbound buffers don’t unfairly compete with write deadlines (#​5750)
• In client and leafnode results cache, populate new entry after pruning (#​5760)
• Use newly-available generic sorting functions (#​5757)
• Set a HTTP read timeout on profiling, monitoring and OCSP HTTP servers (#​5790)
• Improve behavior of rate-limited warning logs (#​5793)
• Use dedicated queues for the handling of statsz and profilez system events (#​5816)

Clustering

• Reduce the chances of implicit routes being duplicated (#​5602)

JetStream

• Optimize LoadNextMsg for wildcard consumers that are consuming over a large subject space (#​5710)
• When sync/sync_interval is set to always, metadata files for streams and consumers are now written using O_SYNC to guarantee flushes to disk (#​5729)
• Walking an entire subject tree is now faster and allocates less (#​5734)
• Try to snapshot stream state when a change in the clustered last failed sequence is detected (#​5812)
• Message blocks are no longer loaded into memory unnecessarily when checking if we can skip ahead when loading the next message (#​5819)
• Don’t attempt to re-compact blocks that cannot be compacted, reducing unnecessary CPU usage and disk I/Os (#​5831)

Monitoring

• Add StreamLeaderOnly filter option to return replica results only for groups for which that node is the leader (#​5704)
• The profilez API endpoint in the system account can now acquire and return CPU profiles (#​5743)

Miscellaneous

• Various improvements to unit tests (#​5668, #​5607, #​5695, #​5706, #​5825, #​5834)

Fixed

General

• Fixed a panic when looking up the account for a client (#​5713)
• The ClientURL() function now returns correctly formatted IPv6 host literals (#​5725)
• Fixed incorrect import cycle warnings when subject mapping is in use (#​5755)
• A race condition that could cause slow consumers to leave behind subscription interest after the connection has been closed has been fixed (#​5754)
• Corrected an off-by-one condition when growing to or shrinking from node48 in the subject tree (#​5826)

JetStream

• Retention issue that could cause messages to be incorrectly removed on a WorkQueuePolicy stream when consumers did not cover the entire subject space (#​5697)
• Fixed a panic when calling the raftz endpoint during shutdown (#​5672)
• Don’t delete NRG group persistent state on disk when failing to create subscriptions (#​5687)
• Fixed behavior when checking for the first block that matches a consumer subject filter (#​5709)
• Reduce the number of compactions made on filestore blocks due to deleted message tombstones (#​5719)
• Fixed maximum messages per subject exceeded unexpected error on streams using a max messages per subject limit of 1 and discard new retention policy (#​5761)
• Fixed bad meta state on restart that could cause deletion of assets (#​5767)
• Fixed R1 streams exceeding quota limits (#​5771)
• Return the correct sequence for a duplicated message on an interest policy stream when there is no interest (#​5818)
• Fixed setting the consumer start sequence when that sequence does not yet appear in the stream (#​5785)
• Connection type in scoped signing keys are now honored correctly (#​5789)
• Expected last sequence per subject logic has now been harmonized across clustered stream leaders and followers, fixing a potential drift (#​5794)
• Stream snapshots are now always installed correctly on graceful shutdown (#​5809)
• A data race between consumer and stream updates has been resolved (#​5820)
• Avoid increasing the cluster last failed sequence when the message was likely deleted (#​5821)

Leafnodes

• Leafnode connections will now be rejected when the cluster name contains spaces (#​5732)

Complete Changes

v2.10.18

Compare Source

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.22.5

Dependencies

• github.com/nats-io/jwt v2.5.8 (#​5618)
• github.com/minio/highwayhash v1.0.3 (#​5627)
• golang.org/x/crypto v0.25.0 (#​5627)
• golang.org/x/sys v0.22.0 (#​5627)

Improved

Embedded

• Export server function to initiate “lame duck mode” when embedding NATS (#​5660)

JetStream

• CPU spike during recalculation of first message in the memory store (#​5629)

Fixed

JetStream

• Fix duplicate callbacks on full wildcard match (#​5610)
• Multiple fixes for the filestore per-subject state (#​5616)
• Fix checkSkipFirstBlock which could return a negative index if the first block in the per-subject index is outdated (#​5630)
• Don't ack messages if consumer is filtered and they were not applicable (#​5639, #​5612, #​5638)
• Protect against possible panic in the filestore where the stree index is nil (#​5662)
• Prevent panic when shutting down a server immediately after starting it (#​5663)

Complete Changes

v2.10.17

Compare Source

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.22.4 (#​5487)

Dependencies

• golang.org/x/sys v0.21.0 (#​5508)
• golang.org/x/crypto v0.24.0 (#​5509)
• github.com/klauspost/compress v1.17.9 (#​5538)
• github.com/nats-io/nats.go v1.36.0 (#​5538)

Added

Monitoring

• Experimental /raftz monitoring endpoint for retrieving internal Raft group state for diagnostic purposes (#​5530)

Improved

Core

• Reorder struct fields in stree for improved memory alignment (#​5517)

JetStream

• Improve performance of calculating num-pending and interest state of a stream (#​5476)
• Improve leadership change signaling (#​5504, #​5505)
• Improved memory-based stream behavior during server restarts (#​5506)
• Reset election timer when leaving observer mode enabling quicker leadership hand-off (#​5516)
• Ensure ack processing is consistent and correct between leader and followers for replicated consumers (#​5524)
• Use per-subject info to speed up load-last filestore operations with wildcard filters (#​5546)
• Populate missing per-subject info after skipping blocks when calculating filtered pending (#​5545)
• Reduced time taken to process consumer deletes when there is a large gap between the consumer ack floor and the stream last sequence (#​5547)
• No longer retrieve the WAL state unnecessarily when installing Raft snapshots (#​5552)
• Increased filestore block and per-subject info cache expiry times to help improve performance on sparse streams (#​5568)
• Reduce allocations in isMatch in filestore/memstore (#​5573)
• Improved handling of out-of-date first blocks in per-subject info entries (#​5577)
• Use stree for message block subject indexing instead of hashmaps (#​5559)
• Avoid loading last message blocks on LoadNextMsg miss (#​5584)
• Add node48 node size to stree, providing better memory utilisation for some subject spaces (#​5585)
• Logging message when exceeding JetStream account limits now prints the account (#​5597)

Monitoring

• Rate-limit statsz updates which reduces load for very large clusters (#​5470, #​5485) Thanks to @​wjordan for the report and contribution!

Changed

MQTT

• Do not wait for JS responses when disconnecting the session (#​5575)

Fixed

Accounts

• Import/export cycle detection (#​5494) Thanks to @​semakasyrok for the contribution!
• Allow callout users to be revoked (#​5555, #​5561)
• Fixed a data race when updating payload limits from JWT claims (#​5593)

Core

• Allow client kick to also kick leafnode connections (#​5587)
• Fix imports sometimes not being available for a client after server restarts (#​5588, #​5589)

JetStream

• Avoid panic on corrupted TAV file (#​5464)
• Performance regression in LoadNextMsg with very sparse or no messages (#​5475)
• Stepdown candidate when append-entry is ahead of last log term (#​5481)
• Fix possible redelivery after successful ack during rollout restarts (#​5482)
• Fix returning maximum consumers limit reached on some consumer updates (#​5489)
• Fix last sequence stream reset on server restart (#​5497)
• Fix data between creating a consumer and determining cluster state (#​5501)
• Prevent interleaving of setting/unsetting observer states (#​5503)
• Fix accounting for consumers with a different replication factor than the parent stream (#​5521)
• Fix potential segfault if mset.mirror was nil when calculating the last loaded message (#​5522)
• Follower stores no longer inherit the redelivered consumer delivered sequence which could break ack gap fill (#​5533)
• Direct Raft proposals will no longer bypass the internal proposal queue which could cause incorrect interleaving of state (#​5543)
• Audit streams that capture $JS.>, $JS.API.>, $JSC.> and $SYS.> subjects are now only allowed if NoAck is set, avoiding potential misconfiguration that could affect the JetStream API, with the exception of the more specific $JS.EVENT.> and $SYS.ACCOUNT.> as these were allowed before (#​5548, #​5556)
• Streams from failed snapshot restores are now cleaned up correctly, fixing potential false positive warnings on /healthz after a failed restore (#​5549)
• Ensure that internal system clients used by Raft groups are always cleaned up correctly, fixing a potential memory leak (#​5566) Thanks to @​slice-srinidhis for the report!
• Fixed a potential panic when updating stream sources on an existing stream (#​5571)
• Fixed panic when creating a stream with an incorrect mapping destination (#​5570, #​5571)
• Fixed returning error when trying to update a stream that has sources with bad subject transforms (#​5574)
• Fixed a bug that would return “no message found” for last_per_subject (#​5578)
• Correctly leave Raft observer state after applies were paused, i.e. due to a catch-up in progress (#​5586)
• JetStream no longer leaks memory when creating and deleting Raft groups (#​5600)
• Fixed a potential panic in consumer ack queue handling (#​5601)
• Fixed data race in runAsLeader (#​5604)

Leafnodes

• Prevent potential message duplication for queue-group subscriptions (#​5519) Thanks to @​pcsegal for the report!

Monitoring

• Ensure consistency of the delivered stream sequence in /jsz filtered consumer reporting (#​5528)

Chores

Config

• Clarify comment on re-use of config Options type for embedded mode (#​5472) Thanks to @​robinkb for the report and contribution!

JetStream

• Added additional memory-based Raft tests (#​5515)

Complete Changes

v2.10.16

Changelog

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

[!WARNING]
A possible regression may result in a server panic at startup when tav.idx files were incorrectly truncated down to zero bytes. You can work around this problem by deleting tav.idx files that are zero bytes in length before starting the server. Zero-byte files could exist as a result of a previous server crash before a successful file sync to disk occurred.

Go Version

• 1.22.3 (#​5438)

Dependencies

• github.com/nats-io/jwt/v2 v2.5.6 (#​5328)
• golang.org/x/sys v0.20.0 (#​5388)
• golang.org/x/crypto v0.23.0 (#​5413)

Added

• Added Left and Right subject mapping operations (#​5337) Thanks to @​sspates for the contribution!
• Add a /expvarz monitoring endpoint (#​5374)

Improved

Accounts

• Change AccountResolver() to use read lock to prevent contention (#​5351)
• Improve muxed routes with large subject space (#​5399)

Gateway

• Outbound may fail to detect stale connection (#​5356) Thanks to @​wjordan for the report!

JetStream

• Optimize stream subject matching implementation (#​5316, #​5324, #​5329, #​5342, #​5353)
• Improve filestore LoadNextMsg performance (#​5401)
• Prevent blocking writes on meta state filestore flush (#​5333)
• Add logging to measure writeFullState and enforceMsgPerSubjectLimit (#​5340)
• Do not hold filestore lock on msg block loads when looking up the first sequence for subject (#​5363)
• Improve various stream sourcing and mirror behaviors and performance (#​5366, #​5372, #​5379, #​5389)
• Increase the compression threshold for Raft traffic (#​5371)
• Put a maximum idle flush time for the filestore (#​5370)
• Updated subject state expiration (#​5377)
• Simplify writing the full state to index.db (#​5378)
• Added in separate last subject timestamp to track access (#​5380)
• Check consumer leader status without locks (#​5386)
• Various Raft improvements with limited to no state (#​5427)
• Improved consumer with AckAll performance of first ack with large first stream sequence (#​5446)
• Various stream catchup improvements (#​5454)

WebSocket

• Improve generating INFO data to send to clients (#​5405)

Fixed

Config

• Fix to properly deal with block scopes in lexer (#​5406, #​5431, #​5436)

JetStream

• Fix potential deadlock if a panic occurs during calculateSyncRequest (#​5321)
• Fix corner cases of subject matching (#​5318, #​5339) Thanks to @​mihaitodor for the report!
• Prevent stepping down for old election terms (#​5314)
• Prevent WAL truncation during catch-up until after peerstate/snapshot check (#​5330)
• Fix various delivery counter logic (#​5338, #​5361)
• Reset election timeout only on granted vote request (#​5315)
• Ensure stream catchup syncs after server crash and restart (#​5362)
• Prevent race condition for mirroring a consumer (#​5369)
• Ensure messages are removed after consumer updates occur on interest-based streams (#​5384) Thanks to @​MauriceVanVeen for the report and contribution!
• Cleanup messages on interest stream after consumer interest changes (#​5385) Thanks to @​tyler-eon for the report and @​MauriceVanVeen for the contribution!
• Fix potential redelivery of acked messages during server restarts (#​5419)
• Hold onto tombstones for previous blocks on compact (#​5426)
• Fixes for rescaling streams with sources (#​5428)

WebSocket

• Fix data races during shutdown (#​5398)

Chores

• Various test improvements (#​5319, #​5332, #​5341)
• Document field names (#​5359)

Complete Changes

v2.10.14

Compare Source

Changelog

(Note there was no 2.10.13 version 🙂)

Refer to the 2.10 Upgrade Guide for backwards compatibility notes with 2.9.x.

Go Version

• 1.21.9 (#​5300)

Dependencies

• github.com/nats-io/nats.go v1.34.1 (#​5271)
• golang.org/x/crypto v0.22.0 (#​5283)

Improved

Auth

• Improve clone behavior to prevent unintended references (#​5246) Thanks to Trail Of Bits for the report!
• Apply constant-time evaluation of non-bcrypt passwords (#​5247) Thanks to Trail Of Bits for the report!

JetStream

• Reduce lock contention when looking up stream metadata (#​5223)
• Optimize matching a subject when applying per subject message limits (#​5228)
• Optimize waiting queue for pull consumers to reduce excessive memory and GC pressure (#​5233)
• Improve error handling in filestore to prevent duplicate nonces being used and ignored errors (#​5248) Thanks to Trail Of Bits for the report!
• Improve interest and workqueue state tracking to prevent stranded messages during concurrent consumer acks and stream deletes (#​5270)
• Introduce store method to push down and optimize multi-filter subject matching used by consumers (#​5274) Thanks to @​svenfoo for the report!
• Various improvements and fixes for clustered interest-based streams and associated consumers (#​5287)
• Return errors and/or adding logging for rare filestore conditions (#​5298)
• When explicitly syncing to the filesystem, hold the message block lock to prevent possible downstream corruption (#​5301, #​5303)

Fixed

OS

• Fix for race checkptr panic on macOS/Darwin on Go 1.22 (#​5265)

Connections

• Address possible memory leak due to connections not be released (#​5244) Thanks to @​davidzhao for the report!

JetStream

• Fix incorrect subject overlapping checks that could lead to multiple consumers or streams bound to the same subjects (#​5224)
• Improve situations that could result in orphan messages in streams (#​5227)
• Protect against corrupt message block when doing indexing (#​5238) Thanks to @​kylemcc for the report!
• Fix consumer config check of max deliver when backoff is set (#​5242)
• Ignore Nats-Expected-* headers from source stream (#​5256) Thanks to @​ramonberrutti for the report and contribution!
• Add missing check that could result an extended purge or compact to fail in memory-based streams (#​5264)
• Fix issue that could result in skipping valid messages when loading them from the filestore (#​5266)
• Use cluster-scoped lock when processing a leader change (#​5267)
• Fix missing unlocks in filestore and streams in certain error conditions (#​5276) Thanks to Trail Of Bits for the report!
• Ensure lock is held for the duration of a filestore truncate (#​5279)
• Fix race condition when checking for the stream interest state (#​5290)
• Ensure dangling NRG directories are cleaned up when the in-memory stream/consumer are deleted (#​5291)
• Perform a standard stream purge when ack floor is higher than the last known state (#​5293)
• Handle concurrent creation of workqueue consumers that could result in overlapping interest (#​5295) Thanks to @​LautaroJayat for the report!
• Fix possible case of multiple deliveries of the same message that cause the delivery count decreasing (#​5305)

Monitoring

• Fix JSZ account filtering behavior when requesting stream details (#​5229)

OCSP

• Fix stapling during gateway reconnect and configuration reload (#​5208)

Chores

• Fix incorrect function names in comments (#​5237, #​5289) Thanks to @​depthlending and @​needsure for the contributions!
• Improve workqueue stream sourcing tests (#​5112)
• Speed-up tests and fixup lint reports (#​5258)
• Improve hardened systemd configuration example (#​5272)
• Refactor LoadNextMsgMulti store tests to guard against drift (#​5275, #​5277)
• Rename orphan messages and replicas test cases (#​5292)

Complete Changes